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ABSTRACT 



The president of the school board of the Chicago Public 
Schools (CPS) Illinois mandated that the Student Assessment unit provide 
training for all CPS teachers on how to help their students score higher on 
standardized tests. This paper describes the two books developed as training 
materials and their preparation. The first step was a literature review that 
culminated in two brief chapters on test-taking skills and improving 
students' attitudes and motivation. Several chapters discuss the skills 
measured on the district and state assessments and provide some general hints 
on preparing students to take tests in these areas. Workshop training 
sessions focus on developing students' thinking skills using item templates 
that help teachers make their own questions to measure thinking skills. The 
overall message of the training sessions and test preparation books is that 
standardized tests require that students apply critical thinking skills, and 
if students are accustomed to answering those types of questions on a regular 
basis in class, they are likely to do better on the tests. It is difficult to 
evaluate the impact of the workshops and training materials. Test scores have 
been going up in the CPS, but it is not possible to determine the causal 
factors. Teachers report seeing the value of teaching and assessing thinking 
skills. (SLD) 
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I've been director of student assessment for the Chicago Public Schools fqr^^^ 
16 years and during that time lots of strange things have come across my-' - 
desk and I've become pretty jaded. However, I was unprepared for a mailing 
that told me how I could "see [my] test results soar” on Illinois' Prairie State 
Achievement Exams. At first glance, this was no different from a lot of 
advertising circulars that I get all the time, but what was so remarkable was that 
this particular flyer was hawking a test preparation product for a test that doesn't 
even exist yet! In fact, not only does the test not exist, but the test specs didn t 
even exist yet. At that time, the state had not yet decided what form the Prairie 
State Exams would take. And despite that, someone was already selling test 
prep materials for it! After thirty years of working for bureaucracies, I’ve become 
quite a connoisseur of the absurd and this was something that impressed even 

me. 

The Chicago Public Schools administers assessments that are high stakes for 
both students and staff. A few years ago the president of our school board 
mandated that our Student Assessment unit provide training for all CPS 
teachers on how to help their students score higher on standardized tests. 

My initial reaction was to cringe. I'd been troubled for many years about 
the unquestioning, unwavering faith some of our teachers and principals have 
in test preparation training. They feel that if they give their students enough 
practice tests, then the kids are sure to do well on the Iowa Tests. And if it 
doesn’t work-which is often the case--it was probably because they 
didn't practice enough, not because the students couldn’t read or do math. 
They would interpret low scores as a sign that they should take away time 
from reading and math instruction so that there can be even more test prep. 

I view many of these activities as little more than anxiety-reduction 
techniques for superstitious grownups and a poor use of time for the kids. My 
colleagues and I did not relish the thought of contributing to that. 

2 

ERIC 



3 



Our task was to figure out how to do this training at our nearly 600 schools^.. . 
and to do it in a way that might actually improve kids’ academic skills and-^--v 
not take time away from content area instruction. Because there are so many 
schools and so few of us, going out to all of our schools wasn't an option, 
so we decided to write two books, Preparing Your Elementary Students to Take 
Standardized Tests (Borger, et al„ 1996a) and Preparing Your High School 
Students to Take Standardized Tests (Borger, et al„ 1996b). Both are available 
at the Chicago Public Schools intranet site (go to http://intranet.cps.k12.il.us, 
click on "Assessments" and then on "Standardized Test Prep ). 

We started by scouring the literature on test preparation and summarizing the 
results in two short chapters on test-taking skills and improving students 
attitudes and motivation. These chapters contain suggestions that will come as 
no surprise to anyone, such as advising the students to follow directions closely, 
budget time appropriately, read all of the answer choices, and check their work 
carefully. We stressed the need to integrate those activities with regular 
instruction, so that test preparation doesn't become a time-consuming add-on. 
There are also several chapters that discuss the skills measured on our district 
and state assessments and provide some general tips about how to prepare 
students to take tests in those areas. For example, in the section on reading, we 

suggest that teachers 

. assist students in developing strategies to use if they re having trouble 
understanding something they’re reading 
. . ask students to retell stories in their own words, communicating the main 

idea, identifying sequence, and providing details 

. have the students read a wide variety of materials and 
. incorporate reading and writing activities in all subject areas. 

| think most of us would agree that these are test prep activities we could 

live with. 



O 

ERIC 



3 



While teaching test-taking skills may have its place, a far greater emphas^^ ■_ 
must be placed on teaching students critical thinking skills. Or to put it 
another way, no amount of test-taking skills can overcome not knowing how to 
read or do math. The real heart of our test prep books is a chapter on assessing 
thinking skills in the classroom. Rick Stiggins observed that the 
overwhelming majority of questions asked in class were recall questions and 
he and his colleagues (Stiggins, Rubel and Quellmalz, 1988) developed 
instructional materials on how to assess thinking skills in the classroom. Over 
the years we've expanded upon those materials and conducted training for 
teachers from all levels from preschool to grade 12. The parts of our books 
deal with measuring thinking skills was closely based on the work of Rick and 

his colleagues. 

The workshops use Edys Quellmalz's taxonomy, which identifies five categories 
of thinking skills: recall, analysis, comparison, inference and evaluation 
(see Figure 1). We make the following points: 

. There is a connection between classroom assessment and standardized 
tests. 

• Assessments include not only tests, but oral questions and homework, t 

. It’s important to assess all students’ thinking skills, regardless of age, 
disability or proficiency in the English language. 

[Insert Figure 1 about here] 

We encourage teachers to create their own items using the keywords in Figure 
1 , the question templates shown in Figure 2, or the CPS Performance 
Assessment Idea Book (accessible from http://intranet.cps.k12.il.us, click on 
-Assessments" and then on "Ideas and Rubrics"). The fill-in-the blank templates 

are especially versatile and easy to use. 

[Insert Figure 2 about here] 
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Here are some examples of item templates: 

. Recall : What is ? 

. Analysis: How does work? 

. Comparison: Compare to — 

. Inference: What do you think would happen if ? 

. Evaluation: In your opinion, what is the best solution to the problem 
? Why is it the best? 




of 



In our workshops we generally ask teachers to use the above templates to write 
questions in each category that assess knowledge of some leisure time 
activity they enjoy; they're told it absolutely can’t have anything to do 
with school. For example, some questions we might create using these 

templates are: 

. What is a beiqnet ? 

. Hnw does ordering breakfast a t Cafe du Monde work? 

. P.nmparft New Orleans to Peoria. 

. What do you think would happen if the AERA annual meeting were heldj n. 
Peoria ? 

• In your opinion, what is the best solution to the problem of papers whose. 
titles contain the words "deco nstructionism," "hermeneutic s, — semiotics or 
"ontological ?" Why is it the best? 

We then ask for volunteers who are willing to share their items with the entire 
group. This helps teachers see that (1) it's easy to generate questions that 
measure thinking skills and (2) the technique may be used across a broad 
range of content areas (though we do find that it's not as useful for math as 
in other subjects). We generally followed this individual activity by a group 
activity in which teachers team up with colleagues who teach the same grade or 
subject so that they can develop questions together. 
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When the test preparation books were first distributed to all of our , • ; . . 

elementary and high school teachers, we went out to as many schools as we- 
could, but with nearly 600 schools, we had no realistic alternative to using a 
trainer of trainers model. We knew that was a less than optimal method, so we 
did what we could to make that method work. Each school was asked to send 
at least one person to a regional training session and nearly all of them did. 

Many people are understandably uncomfortable about attending a brief training 
workshop and then standing up in front of a bunch of their colleagues and 
possibly looking like an idiot, so we tried to make things as painless for 
them as possible. Those who attended our training workshop received a set 
of PowerPoint overheads keyed to either the elementary or high school test 
prep book, complete speaker's notes, handouts, and a set of 18 training 
activities to use with their teachers. An example of one of those activities is 
shown in Figure 3. We walked them through the whole package and had them 
take part in a number of the activities. Staff from our office were there to provide 
moral support and we were also available afterwards for emergency phone 
consultations and troubleshooting. 



[Insert Figure 3 about here] 

The message we were trying to send was that standardized tests require 
students to apply critical thinking skills and if students become accustomed 
to answering those types of questions on a regular basis in class, they are 
likely to do better on the tests. We also pointed out that those questions 
are also likely to make classes more interesting and fun. We tried to 
convince teachers that (1) it's important to ask the kids something other than 
recall questions and (2) a question doesn’t have to be multiple choice to be 
good test preparation, so it's not only okay, but desirable, to ask open-ended 
questions that require students to apply critical thinking skills. We often hear 
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that use of standardized testing promotes memorization and rote learnmg^jj^ 
this has always been puzzling to me, because 1 can't imagine how memorizing 
anything would help a student do better on a standardized reading test. 

Certainly our aim was to get teachers out of the habit of asking mostly recall 

questions. 

The message that it was important to teach and assess thinking skills was 
reinforced in several ways: 

• Information on the categories of thinking skills and assessment keywords 
were included in the lesson plan books that all CPS teachers must use. 

• CPS curriculum materials frequently refer to development of the various 
categories of thinking skills 

. The Illinois State Board of Education used the taxonomy in a number of 
their assessment-related publications for teachers 

• Teachers who write items for our high school course exams are asked to 

write items that assess students' thinking skills. 

. Region staff development meetings usually have sessions on teaching and 

assessing thinking skills. 



So, I hear you wondering, did it work? 

Well, it's hard to tell. Nothing happens in isolation and although our test 
scores have been going up, it's impossible to tell exactly what combination 
of factors caused that. I'd like to report that trade in commercial test 
prep materials has come to a screeching halt in Chicago. That hasn't 
happened-there will always be people looking for what they believe will be an 
easy way out of the problem of low achievement. And the CPS recently had 
a consultant develop special test prep lessons exclusively for use in after-school 
remedial programs. However, I do know that the concentration on assessing 
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thinking skills has had a profound effect at some of our schools. 

of the schools that I knew were always ordering moFe copie : • 



the stories they told were remarkably similar. 

. They conducted training for their teachers using the PowerPoint 

presentations, speaker’s notes, and training activities. They found the 
materials easy to use and appreciated not having to create their own. 

. Teachers received laminated copies of the question templates shown in 
Figure 2 and some got poster-sized versions to hang in their classrooms. 

. They got groups of teachers working together to write items that assessed 
thinking skills. In high schools they met by department and in elementary 
schools, by grade level. Teachers had the opportunity to swap items with 
their colleagues, share successes and troubleshoot problems. In all cases 



the principals provided time for teachers to meet. 

. Lead teachers or administrators encouraged teachers to specifically 



address thinking skills in their lesson plans. 

. Teachers were encouraged to write their own thinking skills items, rather 
than just use whatever questions happened to be in their textbooks. Some 
teachers created question templates that were specific to their own subject 

areas. 

. Some faculty members immediately took ownership of the process. Others 
needed more coaxing, but eventually came on board. A few are still waiting 
for all this to blow over. However, everyone now knows what thinking skills 
are and why they’re important. 

. Efforts were made to make all faculty responsible for teaching and 
assessing students’ thinking skills. 

. Support came from unexpected places. One school had a group of 

enthusiastic art teachers who found ways to incorporate thinking skills into 
their lessons. Teachers in a vocational education program in food service 
assigned students readings from Like Water for Chocolate and then asked 
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students questions that required them to apply critical thinking skills. 

Similarly, a number of health teachers became actively involved in develop ' g 

and assessing students’ thinking skills. 

. All of these schools got students involved in learning what thinking 
skills are, in developing questions, and in critically evaluating their own 

work. 

• Students were asked to justify the answers they gave. 

. Teachers were encouraged to actively engage lower-performing students in 
discussions and assignments to develop their critical thinking skills. 

. Though the schools underwent some profound change, teachers were guided 

in a gentle, non-threatening, non-evaluative way. 

. The process of change was slow. It takes several years, but it’s worth 
waiting for. One assistant principal happily reported that her faculty now 
takes thinking skills instruction and assessment for granted and that their 
faculty is now complaining about other things instead. 

. Student achievement improved. 

A funny thing happened when teachers started assessing thinking skills-they 
realized that they needed to teach thinking skills. Which, of course, was 
exactly what we had in mind from the start. All of these schools had made a 
commitment to teaching and assessing thinking skills. They bought into the 
idea that good teaching is good test preparation. And we learned that it’s 
amazing what you can get people to do if you just call it "test preparation." 
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Question Templates 



Recall 



Note: Any question becomes a recall question 
if the answer has already been explicitly 
provided to the student in class or in the 
text. 

take place? 



Define the term 
What is a 
Who did 
Name 





M What caused 



How does 
Sort these 






What is another possible cause of 

Use the table to determine • 

Use the graph to determine • 

Graph • 

Outline the • 

Based on the written description, draw a 
diagram. 

Draw your own map of without 

tracing or copying. 

Use the map to determine 

In what sequence did 

g rea k down into its component 

parts. 

Give an example of • 



Find examples of [a literary device] in your 
readings. 

Analyze the i n .• 



Classify these 
Separate the _ 

Translate 

Analyze how _ 
Explain how _ 



according to 
from the 



works. 



happen? 



How did the author convey ? 

How is/are . used to 

is this? 



What literary form is being used? 

What technique is being used? 

What information is needed? 

Is the information relevant? 

Into what groups can you organize these? 

Draw a picture that illustrates what is 
described in the story 



What kind of a 
Which one does not belong in this group? 

What is the function of ? 

What is the purpose of ? 

What is the relationship between 

and ? 

Is there a pattern here? What is the pat- 
tern? 

Use manipulatives to illustrate a concept. 

Build a model of • 

Measure . 

What words does the author use to paint 
image of in your mind? 
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Inference 

■ Hypothesize what wilt happen if 

Predict what will happen if 

Apply the rule to .- 

Solve the problem • 

Predict how 




will end. 



Write a research paper on 



What do you think will happen? 

What is the main idea of ? 

What is the overall theme of • 

What is the moral of ? 

Develop a plan to • 

Propose and describe an invention that 
fills some need. 

What is the best title for this reading 
selection? 



Based on your readings, what can you con- 
clude about .- 

What was the author's point of view? 

Solve a logic puzzle. 

What if .? 

What rule applies here? 

What generalization can you make from this 
information? 

Create a • 

Design a • 

What clues are you given about 







worth the costs? Explain 



Was 

your answer. 

Was the argument convincing? What 
makes you think so? 

Did behave appropriately? Why? 

What would you have done in this situa- 
tion? Why? 

Write a critique of .• 

Was this experiment well designed? De- 
fend your answer. 

Decide the best solution to the problem of 
Why do you think so? 

How well are the conclusions supported 
by the data / facts/evidence? Explain. 

Whose argument is stronger? Why do you 
think so? 

Write a letter to the editor stating your 
position on . 

Develop criteria for judging the quality/ 

soundness/ credibility of .• 

Was this real or imaginary? What makes 
you think so? 

Is this fact or opinion? What evidence 
points to that? 



Did 

tion? Give reasons. 

Apply a scoring rubric to this piece of 
work. Explain why you are assigning each 

score. 

What would you have done in this situa- 
tion? Why? 

Review a book, performance or exhibit. 
Justify your evaluation. 

Which is the best? Why dc you 

think so? 

Whose evidence was more convincing? 
Why? 

Whose arguments were most convincing? 
Why? 

If you were the judge, what would your 
decision be? Why? ■ r . 

Give and justify your opinion on 

Write a persuasive essay on • 

What might be a better solution to this 
problem. Explain why it is better. 
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Figure 3 ^ . ; . 

Preparing Your Elementary Students to Take Standardized Tests 



Activity E-1 

Creating HOTS Questions 

Chapters 6-9 contain detailed information on 
the kinds of skills measured on ITBS and 
IGAP. As you can see, the overwhelming 
majority of questions require the student to 
apply higher order thinking skills (HOTS). If 
students are to do well on ITBS and IGAP, it is 
necessary for them to practice answering 
those kinds of questions in everyday 
classroom situations. 

The categories of thinking skills are 
summarized on p. 23. Analysis, comparison, 
inference and evaluation are all considered 
higher order thinking skills. If you have not 
already done so, please take a moment to 
review the chart on that page. Also, take a 
look at the tables of fill-in-the-blank HOTS 
questions shown on pages 25-26. These 
templates or “cookbooks” provide an easy way 
to create questions that assess your students’ 
thinking skills. 

Select a unit you are teaching or a story your 
students are reading (or that is being read to 
them). Using the templates on pages 25-26, 
create HOTS questions that you will ask your 
students: 





See pages 25-26 for recipes for 
HOTS questions. 



• in a class discussion, 

• on a homework assignment, and 

• on a quiz or test. 

Develop questions that assess analysis, 
comparison, inference and evaluation. You 
may find it helpful to use an assessment 
planning chart to organize your questions. A 
blank assessment planning chart may be 
found at the beginning of the Appendix. 

Make a special effort to include HOTS 
questions in all classroom discussions and 
assessments. By doing so, you will give your 
students valuable preparation for ITBS and 
IGAP. As a bonus, the HOTS questions will 
make class discussions more interesting and 
fun for both you and your students. 
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